1. Field of the Invention
The present invention relates to systems and methods for interacting with a mobile device using an external display, and more specifically to providing representations of objects and actions to an external, shared display for a user to select representations and input them to the mobile device.
2. Background of the Invention
Mobile devices must remain small to be portable. This requires innovative forms of interaction designed to work around the limitations of small screens and—although increasing—limited computing power and memory. One attractive way to extend our capabilities to interact with mobile devices is through external and public displays. The price per pixel is falling, and it is not unlikely that shared large displays will spread in public spaces. The human-computer interaction community is becoming increasingly interested in settings where people interact through public displays driven by multi-touch technology. For example, Peltonen et al. studied how crowds of people interacted with a large multi-touch display installed in a central location in Helsinki, Finland, over the course of 8 days. Peter Peltonen, Esko Kurvinen, Antti Salovaara, Giulio Jacucci, Tommi Ilmonen, John Evans, Antti Oulasvirta, and Petri Saarikko. It's mine, don't touch!: interactions at a large multi-touch display in a city centre. CHI 2008: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 1285-1294, New York, N.Y., USA, 2008. ACM.
The contemporary use of such displays is usually characterized as follows: 1) interaction follows the direct manipulation paradigm, i.e., objects on the display resemble objects that are manipulated by interacting with the display through suitable input modalities such as touch; 2) visual feedback is immediate and occurs on the display, which is the primary graphical output medium; 3) the display is driven, and objects are manipulated, by a host computer based on the received input gestures.
This means in particular that all objects manipulated through such a system are shared. Otherwise, the system must have a concept of object ownership, which requires a security policy, authentication, authorization and the delegation of access rights. Anonymous use of such a system for objects that are restricted or for which integrity and authenticity properties must be ensured is difficult. At the very least, all users must trust the display system and its host computer to enforce the security goals of individual users. For a public system under unknown authority, this is unlikely to be achieved.
Multi-touch input gains significant traction and is the technology of choice for multi-user interaction with public displays. In the past, researchers have suggested ways to leverage camera-equipped mobile devices as input modalities for these systems. See Rafael Ballagas, Michael Rohs and Jennifer G. Sheridan; Sweep and Point & Shoot: Phonecam-Based Interactions for Large Public Displays; CHI 2005: Proceeding of the twenty-third annual SIGCHI conference on Human factors in computing systems, pages 1200-1203, Portland, Oreg., USA, 2005; ACM; and Hao Jiang, Eyal Ofek, Neema Moraveji, Yuanchun Shi; Direct Pointer: Direct Manipulation for Large-Display Interaction using Handheld Cameras; CHI 2006: Proceeding of the twenty-fourth annual SIGCHI conference on Human factors in computing systems, pages 1107-1110, Montreal, Quebec, Canada, 2005; ACM. A prototypical architecture of such a system is shown in FIG. 1, showing a system 500 that enables users with a mobile device 502 to interact with applications 504 on a host system 506 using an external display 508. The mobile device 502 recognizes displayed objects 510 and instructs the host computer 506 how to manipulate the objects 510 through simulated dragging and clicking. Alternatively, the device 502 recognizes its own position relative to the display 508 and transmits this position information to the host computer 506, similar to what a mouse would do. This information is then again used by the host computer 506 to perform manipulation tasks. Occasionally, the system allows users to annotate the manipulated object using the mobile device, e.g., by writing a note, as described in S. Greenberg, M. Boyle, and J. Laberge. PDAs and shared public displays: Making personal information public, and public information personal. Personal Technologies, 3(1):53-64, 1999.
Thus, it is desired to develop systems and methods that combine the benefits of using external displays for mobile devices with a secure interaction that minimizes conflicts in a public setting.